Reinforcement Learning Approaches to Flocking with Fixed-wing Uavs in a Stochastic Environment Approches D’apprentissage Par Renforcement Pour Les Essaims De Drones À Voilure Fixe Dans Les Environnements Stochastiques

In the past two decades, unmanned aerial vehicles (UAVs) have demonstrated their efficacy in supporting both military and civilian applications, where tasks can be dull, dirty, dangerous, or simply too costly with conventional methods. Many of the applications contain tasks that can be executed in parallel, thus can benefit in terms of effectiveness from deploying multi-UAVs working together as a force multiplier. However, to do so requires autonomous coordination among the UAVs, similar to swarming behaviors seen in animals and insects. This research looks at flocking with fixed-wing UAVs in the context of a model-free reinforcement learning problem, structured as a Markov decision process. The advantage of a model-free approach is that it can be applied to different platforms without the plant and disturbance models, which implies greater adaptability to changing environments and unforeseen situations. We propose two learning approaches that enable the agents, modeled as small fixed-wing UAVs, to learn control policies that facilitate flocking in a leaderfollower topology, while operating in a non-stationary stochastic environment. The first approach is based on Peng’s Q(λ) with a variable learning parameter, which learns through direct reinforcement learning. The second approach is based on Sutton’s Dyna-Q where on-line learning, model learning, and planning are integrated to improve sample efficiency. Our approaches are compared to existing works by evaluating the respective policies at maintaining the desired flocking behavior according to a cost function. Simulation results demonstrate that with the two proposed learning approaches, the agents are able to learn policies that facilitate flocking with a single leader, more importantly, the agents are able to adapt their policies to non-stationary stochastic environments.